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[57] ABSTRACT 

An information retrieval apparatus features a three-di- 
mensional display unit for displaying a set of indexes 
controlling attributes of a database in a three-dimen- 
sional image space in sight of a user with index display 
images so as to allow the user to visually recognize the 
set of indexes. An input unit detects motion patterns of 
the user's body against the index display images as input 
information and displays the motion patterns in the 
three-dimensional image space. An arithmetic operation 
unit recognizes the input information of the motion 
patterns received through the input unit and performs 
set logical operations of selected images in accordance 
with the motion patterns so as to collate and retrieve 
information from the database. The user can further 
directly operate index display images in a three-dimen- 
sional image space or a two-dimensional image space 
without the necessity of a keyboard and recognize input 
information with his body's motion patterns. The arith- 
metic operation unit performs set logical operations of 
indexes displayed with predetermined index display 
images so as to retrieve desired information from a 
database. Thus, even if the user does not have enough 
experience in operating conventional input and output 
units, such as a keyboard, he can readily and accurately 
retrieve desired objects from a medium such as a data- 
base. 

9 Claims, 20 Drawing Sheets 
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On the other hand, the data gloves are input inter- 

INFORMATION RETRIEVAL APPARATUS faces for inputting the motions of the user's hands into a 

computer. The data gloves are provided with magnetic 
This application is a continuation of application Ser. sensors and optical fiber cables over the surface thereof. 
No. 07/911,480, filed on Jul. 10, 1992, now abandoned- 5 Thus, the user can detect and recognize the positions 

BACKGROUND OF THE INVENTION md motion P attems of ^ hands m6 ^P 11 * particular 

information into the computer. 

1. Field of the Invention As was described above, the conventional informa- 
The present invention relates to an information re- tion retrieval apparatuses with a conventional keyboard 

trieyal apparatus, and in particular to an information 10 and so forth were not user friendly. Even if they were 

retrieval apparatus with artificial reality (AR) where user friendly, they could not be effectively operated. In 

the user can select and hold capsule images that a three- addition, since retrieved information was displayed on a 

dimensional image display unit displays in a three-di- flat display such as a CRT (cathode ray tube) monitor, 

mensional image space with his hands so as to collate the user could not straightforwardly recognize the 

and retrieve information from a database. 15 weighting of retrieval indexes. Further, with the flat 

2. Description of the Related Art display, another display method for displaying a plural- 
In this age of information, flooding, there is a great ity of screens at a time, that is, a multi-window could be 

need for quickly and accurately retrieving desired infor- used. However, as the amount of information increased, 

mation. Thus, the importance of an operation for since the display screens were overlapped, the visibility 

readily retrieving a great amount of information is in- 20 deteriorated, and the correlations between each infor- 

creasmg. mation of retrieval indexes could not be effectively 

Input and output (I/O) interfaces such as a keyboard, displayed, 

a touch panel, and a mouse for use in the conventional On the other hand, the virtual reality was used as a 

information retrieval apparatuses are used to input key- simulation tool of computer graphics. However, thus 

words and the like. With keywords being input, infor- far, the virtual reality has not yet been used for designate 

mation is collated and retrieved in the forms of images ing media directly accessing a database stored in a com- 

and hard copies. Thus, thus far, only users who are puter with the data gloves and for performing set logi- 

accustomed to the operations of the keyboard and fa- cal operations. 

miliar with the relevant database system can retrieve 3Q An object of the present invention is to solve the 

desired information by operating the keyboard and so above mentioned problems involved in the related art 

forth of the terminal units. and to provide an information retrieval apparatus with 

On the other hand, in the recent computer simulation which the user can select media as indexes allocated as 

field, user-friendly computer interfaces which inge- images in the image space with his hands, issue prede- 

niously utilize human beings' motions and their five 35 termined commands, and straightforwardly retrieve 

senses such as sight and hearing have been developed information from a database, 
and studied. These interfaces are referred to as artificial 

reality. SUMMARY OF THE INVENTION 

At the present time, the artificial reality can be di- The present invention relates to an information re- 
vided into two major categories. One is referred to as 40 trieval apparatus, and in particular to an information 
virtual reality in which a **user agent" who is an agent retrieval apparatus with artificial reality (AR) where 
of the user of the interface is displayed in computer the user can select and hold capsule images that a three- 
graphics images. The user can experience a world dimensional image display unit displays in a three-di- 
which is spread out in the computer graphics images mensional image space with his hands so as to collate 
through the **user agent." 45 and retrieve information from a database. 

The other category is referred to as tele-existence in An object of the present invention is to solve the 

which the user can operate a robot at a remote position above mentioned problems involved in the related art 

as his agent through a manipulator. While the tele-exist- and to provide an information retrieval apparatus with 

ence focuses on operations under the real environment, - which the user can select media as indexes allocated as 

the virtual reality focuses on the use of the ' t user agent*' 50 images in the image space with his hands, issue prede- 

as an interface of the computer itself. termined commands, and straightforwardly retrieve 

As an example of computer simulation tools for ac- information from a database, 
complishing the above mentioned virtual reality, a The present invention in one aspect thereof provides 
three-dimensional display unit and data gloves have an information retrieval apparatus, comprising a three- 
been developed. The former three-dimensional display 55 dimensional display unit for displaying a set of indexes 
unit provides for a three-dimensional computer graph- controlling attributes of a database in a three-dimen- 
ics world. A variety of three-dimensional display units sional image space in the sight of a user with index 
have been proposed by U.S. Pat. Nos. 4,881,068, display images so as to allow the user to visually recog- 
4,853,769, 4,834,476, 4,160,973, and so forth. In the nize the set of indexes, an input unit for detecting a 
virtual reality, some three-dimensional display units 60 motion pattern of the user's body against the index 
utilize a mechanism where the motions of the user's display images as input information and for displaying 
hands do not intercept the images being displayed so the motion patterns in the three-dimensional image 
that he can feel that he is directly handling three-dimen- space, and an arithmetic operation unit for recognizing 
sional images with his hands. For example, "Eyephone" the input information of the motion patterns received 
which was developed by NASA and produced by VPL 65 through the input unit and for performing set operations 
Research Company (United States) is a three-dimen- of indexes displayed with predetermined index display 
sional^iisftaymut which is in a ski goggle shape and has images so as to collate and retrieve information from the 
(anJKCD display device inside thereof. database. 
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The present invention in another aspect thereof pro- 
vides an information retrieval apparatus, comprising a 
three-dimensional display unit for displaying a set of 
indexes controlling attributes of a database in a three-di- 
mensional image space in the sight of a user with index 5 
display images and for allowing the user to visually 
recognize the set of indexes with operation capsule 
images, an input unit for detecting motion patterns of 
the user's body against the index display images and the 
operation capsule images as input information and for 10 
displaying the motion patterns in the three-dimensional 
image space, and an arithmetic operation unit for recog- 
nizing the input information of the morion patterns 
received through the input unit and for performing set 
operations of indexes displayed with predetermined 15 
index display images by using a logical operation means 
represented with the arithmetic operation capsule im- 
ages so as to collate and retrieve information from the 
database. 

The present invention in a further aspect thereof 20 
provides an information retrieval apparatus, comprising 
a two-dimensional display unit for displaying a set of 
indexes controlling attributes of a database in a two-di- 
mensional image space in the sight of a user with index 
display images so as to allow the user to visually recog- 25 
nize the set of indexes, an input unit for detecting mo- 
tion patterns of the user's body against the index display 
images as input information and for displaying the mo- 
tion patterns in the two-dimensional image space, and 
an arithmetic operation unit for recognizing the input 30 
information of the motion patterns received through the 
input unit and for performing set operations of indexes 
displayed with predetermined index display images so 
as to collate and retrieve information from the database. 

The present invention is an information retrieval 35 
apparatus, comprising a display unit for displaying a set 
of indexes controlling attributes of a database with 
index display images, an input means for inputting input 
information for causing predetermined index display 
images displayed on the display unit to perform prede- 40 
termined motion patterns, and an arithmetic operation 
unit for recognizing input information through the input 
means and for performing set operations of indexes 
displayed with the index display images so as to collate 
and retrieve information from the database. 45 

According to the present invention, the user can 
directly operate index display images in a three-dimen- 
sional image space or a two-dimensional image space 
without the necessity of a keyboard and recognize input 
information with his body's motion patterns. An arith- 50 
metic operation unit performs set logical operations of 
indexes displayed with predetermined index display 
images so as to retrieve and collate desired information 
from a database. Thus, even if the user does not have 
enough experience in operating the conventional input 55 
and output units such as a keyboard, he can readily and 
accurately retrieve desired objects from an information 
medium such as a database. 

In addition, since a set of indexes for use in collating 
and retrieving information are displayed as index dis- 60 
play images in the three-dimensional image space or the 
two-dimensional image space, the user can clearly and 
visually recognize the indexes for use in retrieving in- 
formation, thereby making the information retrieval 
operation easy. 65 

Further, since a variety of motion patterns of the user 
corresponding to index display images have been de- 
fined as particular input information, the retrieval appa- 



ratus can perform predetermined operations in accor- 
dance with the motion patterns. Thus, the user can 
readily select and discard indexes for use in retrieving 
information by using the input information. For exam- 
ple, by holding and flipping any index display image, 
the user can delete the corresponding retrieval index. 
By moving the index display image inwardly, the user 
can obtain the retrieval index. 

Furthermore, index display images are displayed on 
the display unit. When input information for causing an 
index display image to perform a predetermined opera- 
tion pattern is input through the display unit and set 
logical operations of indexes are performed by an arith- 
metic operation unit, information can be collated and 
retrieved from the database. 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other objects, features and advantages of 
the present invention will become more apparent in the 
light of the following detailed description of best mode 
embodiments thereof, as illustrated in the accompany- 
ing drawings, wherein 

FIO. 1 is a schematic diagram virtually showing the 
relation between an embodiment of a three-dimensional 
image space displayed by an information retrieval appa- 
ratus in accordance with the present invention and a 
user thereof; 

FIG. 2 is a schematic diagram conceptually showing 
a three-dimensional image space displayed by the infor- 
mation retrieval apparatus of FIG. 1; 

FIG. 3 is a schematic diagram showing an example of 
a motion pattern of the user's hand; 

FIG. 4 is a schematic diagram showing an example of 
a motion pattern of the user's hand; 

FIG. 5 is a schematic diagram showing an example of 
a motion pattern of the user's hand; 

FIGS. 6(a) and 6(b) are schematic diagrams showing 
examples of lotion patterns of the user's hand; 

FIG. 7 is a schematic diagram showing an example of 
a motion pattern of the user's hand; 

FIG. 8 is a schematic diagram conceptually showing 
time coordinate axis T in a three-dimensional image 
space; 

FIG. 9 is a schematic diagram conceptually showing 
time coordinate axis T in a three-dimensional image 
space; 

FIGS. 10 (c), 10 (b), and 10 (c) are schematic dia- 
grams conceptually showing time coordinate axis T in a 
three-dimensional image space; 

FIG. 11 is a schematic diagram hierarchically show- 
ing an example of the conception of an encyclopedia 
retrieval system of an embodiment of the present inven- 
tion; 

FIGS. 12(a), 12(6) and (c) are schematic diagrams 
showing information retrieval sequences of the encyclo- 
pedia retrieval system of FIG. 11; 

FIGS. 13 (a), 13 (b) and 13 (c) are schematic diagrams 
showing the information retrieval sequences of the en- 
cyclopedia retrieval system of FIG. 11; 

FIGS. 14 (a) and 14 (b) are schematic diagrams show- 
ing information retrieval sequences for the content of 
the retrieval result of FIG. 13 (c) by using the time 
coordinate axis T; 

FIGS. 15 (a) and 15 (b) are schematic diagrams show- 
ing an example of a set logical operation; 

FIG. 16 is a schematic diagram showing an example 
of a set logical operation; 
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FIG. 17 is a schematic diagram showing an example Next, the situation of the three-dimensional image 

of a word and phrase search operation in accordance space displayed before the eyes of the user 90 will be 

with another embodiment of the present invention; described in detail. 

FIG. 18 is a schematic diagram showing a character In FIG. 2, a frame 2 virtually represents an outline of 

describing operation in a three-dimensional image space 5 a three-dimensional image space in which information is 

in accordance with another embodiment of the present displayed. The three-dimensional image space is set in 

invention; the range where the virtual arms 8R and 8L of the user 

FIG. 19 is a schematic diagram showing a pressure 90, who wears the data gloves 92R and 92L on his 

stimulating mechanism build in a data glove; hands, can amply reach. In the three-dimensional image 

FIG. 20 is a schematic diagram showing a private eye 10 space, capsule images shaped like rectangular cards 3a, 

that the user wears on his face; and ^ Zb, 3c, Zd, and Ze; capsule images shaped like barrels 4a, 

FIG. 21 is a schematic diagram showing a flat display 4b t 4c, 4d, 4e, 4/, and 4# capsule images shaped like 

screen which displays capsule images, '^-======^ oblate spheres 5a, 5b, 5c, Sd, 6a, 6b, 6c, 6d r 6e t and 6J\ 

FIG. 22 is a schematic diagram showing another and spherical capsule images la and lb are displayed, 

display screen which displays capsule images. 15 For example, the rectangular shaped capsule images 

DESCRIPTION OF PREFERRED T f^^ ™* 3 % 3 *' 3 *; 3 * *** 3 f ^ 

EMBODIMENTS resent a set of retrieval indexes of small personal infor- 
mation such as a directory. The retrieval indexes are 

An embodiment of the information retrieval appara- sorted outwardly in the alphabetical order. The same 

tus of the present invention will now be described in 20 level capsule images 4a, 46, 4c, and 4g are displayed in 

accordance with a practical information collation and the same depth. The next level capsule images 4d and 4e 

retrieval sequence. are displayed in a deeper position than the capsule im- 

FIG. 1 is a schematic diagram virtually showing the ages 4a, 4b t 4c, and 4g. The further lower level capsule 
relation between an embodiment of a three-dimensional image 4f is displayed in a further deeper position than 
image space displayed by an information retrieval appa- the capsule image 4d and 4c. Likewise, the capsule im- 
ratus in accordance with the present invention and a ages shaped like oblate spheres 5a, Sb, 5c, Sd t 6a, 6b, 6c, 
user thereof from a third person's point of view. In the 6d, 6e, and 6f are weighted in accordance with the im- 
figure, reference numeral 90 is a user who retrieves portance of the information or retrieval indexes. The 
information from a database using the information re- 3Q spherical capsule images la and 7b displayed at the right 
trieval apparatus of the present invention, The user 90 end of the frame 2 can be used as process capsules repre- 
wears a three-dimensional display unit 91 in a ski goggle senting the selection method and the storage area of 
shape on his face. In addition, the user is fitted with information such as logical operations. The capsule 
(wears) data gloves 92R and 92L on his two hands. The images 3a, 3$, . . . , 6e, 6f, which represent a set of in- 
data gloves 92R and 92L are an example of an input 35 dexes, are index display images. On the other hand, the 
unit The data gloves 92R and 92L have signal lines 93R capsule images la and lb, which represent arithmetic 
and 93L^ respectively. The signal lines 93R and 93L are operation information, are arithmetic operation cap- 
connected to an arithmetic operation unit 94. The arith- sules. 

metic operation unit 94 generates particular input infor- The retrieval indexes represent indexes for use in 
mation data or process commands by using motion ^ collating and retrieving information from the database 
patterns of the user 90 which are detected through the 101. Generally, the retrieval indexes are r eferred to as 
data gloves 92R and 92L. In addition, the arithmetic keywj ffis t kgyjten>s and so forth. The retrieval indexes 
operation unit 94 sends to the signal lines 95R and 95L " " are constructed^ cTattributes retrieved from information 
signals for feeding back motions of the user 90 which stored in the database 101. Thus, in the database re- 
are detected by the data gloves 92R and 92L and mo- 45 trieval, by collating the retrieval indexes with the infor- 
tions of capsule images in the three-dimensional image mation indexes, a set of required information can be 
space. A retrieval arithmetic operation unit 96 is con- extracted. At that point, by linking a plurality of re- 
nected to a database 101 whose attributes are controlled . trieval indexes by using logical operations, more accu- 
with indexes. As shown in FIG. 2, the three-dimen- rate information can be obtained. Thus, proper retrieval 
sional display unit 91 displays the motions of the data so indexes should be readily selected. In addition, logical 
gloves 92R and 92L as virtual arms 8R and 8L in the operations should be performed without restrictions, 
image space so as to form a world of artificial reality in To satisfy these conditions, the information retrieval 
which the virtual arms 8R and 8L can handle a variety apparatus according to the present invention performs 
of capsule images 4, 5, and 6 displayed in the image this retrieval operations by using the above mentioned 
space. In other words, before the eyes of the user 90, a 55 capsule images, 

three-dimensional image space as shown in FIG. 2 is With reference to FIG. 2, the selection operation of 

displayed. retrieval indexes by using the capsule images 3, 4, 5, 6, 

In the three-dimensional image space, the user 90 . . . , and so forth will now be described, 

holds a capsule image 4g with his right hand data glove In FIG. 2, reference numeral 8R is the right, hand of 

R and attempts to flip a capsule image 4c with his left 60 the virtual arms. As shown in FIG. 1, the virtual arm 8R 

hand data glove 92L. In this case, an image where the is a virtual image which can perform the same motion 

virtual right hand 8R attempts to hold the capsule patterns as the real hand of the user such as moving in 

image 4g and the virtual left hand 8L attempts to flip the a three-dimensional image space, opening and closing 

capsule image 4C is displayed before the eyes of the user the hand by input information. FIG. 2 shows the situa- 

90. 65 tion where the thumb and the index finger of the right 

In FIG. 1, reference numeral 7 is a storage area. The hand 8R of the virtual arms 8 hold the capsule image 4g. 

storage area 7 has the shape of a box. The storage area By a motion pattern against the capsule image 4g, a 

7 can accommodate a variety of capsules. retrieval index for a set logical operation shown by the 
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capsule image 4g in accordance with a motion pattern metic operation commands for the retrieval indexes can 

thereof is selected. be input to the arithmetic operation unit. 

In this embodiment, a holding motion was exempli- Next, definitions of motion patterns of the hands will 

fied. However, as input motions, a capsule image can be be exemplified. 

held by the entire right hand 8R of the virtual arms or 5 FIG. 3 is a schematic diagram showing a "gripping*' 

lightly touched by any finger of the right hand 8R. motion for gripping the capsule image 4 with the palm 

In addition, according to this embodiment, when the of the virtual arm SR. At that point, the coordinates of 

user holds the capsule image 4g with the right hand 8R, the position of the capsule image 4 are tracked by the 

he can recognize the status of the information being computer. The state where the virtual arm 8R grips the 

selected with a changed color of the capsule image 4g, 1° capsule image 4 is recognized when the coordinates of 

a selection acceptance sound, or a flashing light Thus, capsule image 4 (A), the coordinates of sensor (B) dis- 

the user 90 can readily sort retrieval indexes and have a posed at the end of the index finger, and the coordinates 

caution and a warning to an incorrect operation with a °f sensor (C) disposed at the end of the thumb are 

sound and a light aligned in the sequence of (A)-{B)-(Q- At this point, in 

Moreover, when the user 90 holds the capsule image 15 ^ embodiment, the retrieval index of the capsule 

Ag with the virtual right hand 8R, he can have a pres- jrnage 4 gripped within the palm is extracted for collat- 

sure on his real right hand. In other words, as shown in ing and retrieving information. 

FIG. 19, the data glove 92R is provided with a pressure F * a 4 * a schematic diagram showing a "holding" 

exerting mechanism 105. The pressure exerting mecha- motion. When the distance between the sensor disposed 

nism 105 is activated in accordance with information 20 at ^ e cnd of ^ e mdex fi n & er ^at of tne thumb is 

being input from the real right hand of the user 90 to the lcss than a predetermined value and a capsule image of 

data glove 92R. a keyword is present between these sensors, the state for 

For example, when the arithmetic operation unit 94 hoIdin g the capsule image is recognized. Alternatively, 
receives input information from the data glove 92R and by P roccssm S a three-dimensional image, it can be re- 
determines that the virtual right hand 8R holds the 25 ogm ^ d " . t 

capsule image 4*, the pressure exterting mechanism 105 F f G * a Mastic diagram showing a "flipping" 

of the data glove 92R is activated and an actual pressure motK>n : When * c motlon ***** or velocit y of the 

is transferred to the real right hand of the user 90. Thus, ^f^P detected by the relevant sensor disposed at the 

the user 90 can positively have a "holding" sense ™ en ? ° f ****** fm S er * predetermined value 

As a practical example of the pressure exerting me- 30 £ d ^r interferes with any capsule image, 

chanism 105, a method where a force applied to each * e sta . te therein the retrieval mdex of the capsule 

finger is detected by a respective distortion gauge and Trf^! hT^' « * 

converted into an air pressure applied to a respective a SS^^ \ are scheme diagrams showmg 

inner balloon can be used. „ * selectmg mouon. In FIG 6 (a), when the sensors 

capsule is changed depending on the amount of infor- ^„<.„i^ a * * I - It/ A, , V " v 

.nation of a rekvant retrieval index, the user 90 can Ae™LTf™ • * ^ "2% ° f 

instantly and visually determine the sfce of the informa- JlSS^'Sf * Isco ^ ed - Itt ad * t,on ' 

tion * as shown m FIG. 6 (b), any capsule images can be se- 

rin iU* u^a *^ i & v j oi c xt_ , 40 lected fr° m those surrounded by both hands. 

fl ™«£™ ^ 8 . L .° f ^ ^ FIG.7showsanexampkofasetlogicaloperation.In 

X£,^ G h n TT <S ^irJlu ^ whe » ^ hand 8R holds Oe capsule 

P ^f TS** T- W '* ^ge 4 and the left hand 8L holds the capsule image i, 
St of T£ZT? n ? capsule nmge 4c ^ ^ wherein ^ AND ^ of » oih 
out of the screen. In this example a practical flipping 45 indexes is performed is recognized. At this point, when 
^to H^lT- a T er ' Wh f ^ ^f, 9 ? at J * e hand 8R picks (hrfdTthe capsSe image 4 
« 7 ^ T** hC ""•."E baCk whieh 15 a set of ««*vJ indexes to be retrieved (or a 
^m^f^H »? P ^ "t.JT OT J° ld the C * P " P luralit y of ^ges). a predetermined logical 

^S£w 5? *T*£ °* , f amC * fc «w t«« "trieval index of the right handSR 

Further, like keywords, the contents of the above 50 can be executed 
mentioned retrieval indexes can be displayed on capsule For example, when the right hand 8R holds the cap- 
images. The mformation displayed with keywords can sule image 4 and the left hand 8L holds the capsule 
be remeved. In this case, for example, by holding two image 5, the "OR" operation of the index of the capsule 
capsule images representing keywords with the fingers image 4 and the index of the capsule image S can be 
one after the other, information relevant to the two 55 performed. Alternatively, under the assumption that the 
keywords can be immediately displayed. This informa- operation capsule image 7a represents an "AND" logi- 
tion retrieval method is particularly effective in the cal operation means, when the user 90 contacts or over- 
event that the user 90 attempts to retrieve complicated laps the capsule images 4 and S held by the right hand 
information relevant to a plurality of keywords. 8R with the operation capsule image 7a held by the left 

With reference to FIGS. 3 to 7, an example of motion 60 hand 8L one after the other, the "AND" operation of 
patterns of the hands recognized by the data glove 92 the indexes of the image capsules 4 and S can be "per- 
will now be described. formed". 

Generally, the hands of the human beings can per- In addition, particular hand motion patterns such as 
form several motions such as holding, picking, flipping, for stopping a process during retrieval execution can be 
selecting, and moving objects. By using capsule images 65 defined. In this case, by designating unusual hand mo- 
as objects, set logical operations for retrieval indexes tions, incorrect motions can be prevented, 
displayed with capsule images can be performed in In the above mentioned motion patterns, the use of 
accordance with a variety of motion patterns, and arith- the data gloves was exemplified. However, it should be 
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appreciated that any input unit which can keep track of 
hand motion patterns can be used regardless of what 
construction die input unit has. Moreover, it should be 
understandable that besides the eyephone, any three-di- 
mensional display unit can be used. 5 

Next, an effective retrieval method in the case where 
retrieval indexes of capsule images contain time factors 
will be described. 

FIG. 8 shows a linear time coordinate axis T dis- 
played in a three-dimensional image space. In the figure, 10 
the capsule image 4 is displayed in such a way that it 
floats along the time coordinate axis T. Since the cap- 
sule image 4 contains a time factor as a retrieval index, 
data corresponding to the time coordinate axis T at 
which the capsule image 4 is present can be retrieved 15 
from the relevant database. For example, when the 
capsule image 4 is present at a position of June, 1990 on 
the time coordinate axis T, only data with respect to 
June, 1990 is extracted from data which is hit with the 
retrieval index of the capsule image 4. In the case where 20 
the user 90 wants to obtain data with a particular range 
of the time factor of this capsule image, when he inter- 
cepts the time coordinate axis T with both the hands 8R 
and 8L of the virtual arms as shown in FIG. 9, the range 
(from Tl to T2) can be designated. Thus, for example, 25 
data in the range from June, 1989 to December, 1990 
can be extracted as the time factor of the relevant cap- 
sule image from the database. 

However, on this linear time coordinate axis T, the 
depth of the axis is linearly proportional to the elapse of 30 
years. Thus, when the user 90 retrieves data which 
relates to old years of the time coordinate axis T, he 
would have difficulty in moving his hands. 

To overcome this drawback, as shown in FIG. 10 (a), 
by arranging the time coordinate axis T in a spiral 35 
shape, the direction of depth of the image space can be 
effectively used. In this case, the time steps of the time 
coordinate axis T can be varied in such a way that the 
old years are coarsely allocated. Thus, years can be 
effectively allocated on the time coordinate axis T. In 40 
FIG. 10 (a), when the user 90 holds the capsule image 4 
with his left hand 8L, he can move the capsule image 4 
along the time coordinate axis T. In addition, for exam- 
ple, by drawing a circle several times with the left hand 
8L, the user 90 can move the capsule image 4 along the 45 
time coordinate axis T, Further, as shown in FIG. 10 
(6), the time coordinate axis T can be formed in a zigzag 
shape. Furthermore, as shown in FIG. 10 (c), the time 
coordinate axis T can be formed in a snaky shape. 

As described above, according to the present inven- 50 
tion, since the user 90 can straightforwardly select de- 
sired information from a plurality of capsule images 
displayed in a three-dimensional image space with his 
hands, the present invention can be applied to a variety 
of information retrieval systems which are currently 55 
available such as encyclopedia information retrieval 
systems, newspaper information retrieval systems, base- 
ball information retrieval systems, patent information 
retrieval systems, historical information retrieval sys- 
tems, map information retrieval systems, medical infor- 60 
mation retrieval systems, telephone directories, com- 
pany organization information retrieval system, and 
personal information retrieval systems. 

Next, with reference to FIG. 11, a practical informa- 
tion retrieval method of an encyclopedia information 65 
retrieval system will be described. 

FIG. 11 is a schematic diagram conceptually showing 
a retrieval method in which the user hierarchically 
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selects a desired conception from a large category to a 
small category. 

First, capsule images 11a, 116, and 11c which are in 
level 1 as large category information are displayed in 
the above mentioned three-dimensional image space. At 
this point, the user 90 will directly hold the capsule 
image 116 representing "history" which is the desired 
information from the capsule images 11a, lib, and 11c in 
the above mentioned manner. Then, the present screen 
will be changed and capsule images 13a, 136, and 13c 
which are in level 2 will be displayed. At this point, the 
user 90 will select the capsule image 136 representing 
"region" which is the desired information in the same 
manner as at the level 1 from the capsule images 13a, 
136, and 13c. Then, the present screen will be changed 
as in the level transition from the level 1 to the level 2. 
Then, capsule images 15a, 156, and 15c which are in 
level 3 will be displayed. At this point, the user 90 will 
select the capsule image 156 representing "Europe" 
from the capsule images 15a, 156, and 15c. Likewise, the 
present screen will be changed. Then, capsule images 
16a, 166, and 16c in level 4 will be displayed. At this 
point, when the user 90 selects the capsule image 166 
representing "Germany" which is the desired informa- 
tion, the present screen will be changed and a capsule 
image representing for example "medieval ages" as a 
selection item of Germany historical information will be 
displayed. Thus, by selecting these capsule images one 
after the other, the user 90 can obtain the desired infor- 
mation. This operation for selecting retrieval indexes 
can be executed in a very short period of time. 

In the above described retrieval method, the screen 
of each level was changed one after the other. How- 
ever, it should be understandable that capsule images 
can be selected in other ways. Although the level 1, 
which is the largest category, should be selected first 
because it is the base position of the information re- 
trieval operation, the display sequence of capsule im- 
ages of the level 2 or lower levels is not limited. For 
example, in any of the following three manners, the user 
90 can reach the desired conception. 

Sequence 1: History — ^region —►Europe — ^German 
—►ages —►medieval ages — ►religion 

Sequence 2: History —►ages —►medieval ages —►region 
—►Europe —►German —►religion 

Sequence 3: History —►religion — ►region —►Europe 
-►ages —►medieval ages 

Thus, capsule images in a plurality of levels (for ex- 
ample level 2 or lower levels) can be displayed at a time 
as shown in FIG. 11. Alternatively, by grouping cap- 
sule images through a plurality of selection screens, 
those in the level 1 only or those in the level 2 only can 
be displayed. 

Next, with reference to FIGS. 12 and 13, an opera- 
tion method that the user 90 can use will be described in 
detail. 

In these figures, the retrieval is sequence represented 
with STEP I to STEP 6. In accordance with these 
steps, the operation method will be described. STEP 1: 

The user 90 will select a capsule image representing 
the desired conception from capsule images in the level 
1, which is the largest category. These capsule images 
are displayed in such a way that they are floating in the 
three-dimensional image space. Then, the user 90 will 
select and hold a desired capsule image representing the 
conception to be retrieved with, for example, the left 
hand 8L. Thereafter, the user 90 will move the capsule 
image held with the left hand 8L to a storage area 7 
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disposed on the right of the three-dimensional image readily retrieve information with respect to time factors 

space (see FIG. 12 (a)). along the time coordinate axis T. 

STEP 2: In the case where information to be retrieved does 

The user 90 will select and hold a capsule image not contain time factors, a "coefficient of importance" 

representing the desired conception from capsule im- 5 can be applied to the time coordinate axis T so as to 

ages in the level 2 with his left hand 8L. Thereafter, the display weighted information in accordance with the 

user 90 will move the capsule image held by the left positions along the time coordinate axis T. In the case 

hand 8L to the storage area 7 (see FIG. 12 (6)). where there are several sets of relevant information, 

STEP 3: when the user 90 holds particular information, the rele- 

The user 90 will select and hold a capsule image 10 vant information can be redisplayed. Thus, even if a 

representing the desired conception from capsule im- great amount of information is handled, the user 90 can 

ages in the level 3 with his left hand 8L. Thereafter, the straightforwardly handle it without confusion, 

user 90 will move the capsule image held with the left Ncxt > reference to FIGS. 15 and 16, a retrieval 

hand 8L to the storage area 7 (see FIG. 12 (c)). method for selecting information with set logical opera- 

STEP 4: 15 tions by using capsule images in the same level will be 

The user 90 will select and hold a capsule image described, 

representing the desired conception from capsule im- nG - 15 ( a ) shows Ae ^ where ^ 90 ^ 

ages in the level 4 with his left hand 8L. Thereafter, the seIect a <»P™k image 50 representing for example 

user 90 will move the capsule image held with the left "United States" in the same situation of the STEP 4, 

hand 8L to the storage area 7 (see FIG. 13 (a)). 20 move the capsule image 50 to a storage area 51, and 

STEP 5* select and move a capsule image 53 representing "Ja- 

The user 90 will select and hold a capsule image and a capsule image 52 representing "Germany" 

representing the desired conception from capsule im- m 5 ^ Th ^f? ftcr ' 83 m ** STEP 6 m 

ages in the level 5 with his left hand 8L. Thereafter, the „ *J G - 13 (c) ' ^ user 9 ° ^ ^ ^necessary images out 

user 90 wfll move the capsule image held with the left 25 f°?& » * ma f^ **, m ? C 

hand 8L to the storage area 7 (see FIG. 13 (*)). ™ 6 sh ™ n m FIG \ ? < c > ?° 83 t0 ^ 

STEP 6* three capsule images which are the capsule image 50 

Next, the user 90 will flip unnecessary capsule images ******* "Vmted States", the capsule image 53 rep- 
out of the storage area 7 with the left hand 8L (see FIG. 3Q »»g^ 

^M^S^ 90 ^ ^ remaining ^ ovm m HG . 15 (b)j ^ mer ^ will 

ri^ 

me user >u nips tne unnecessary capsule images out of 35 siand 53 against ^ ch other . ^ 

motion represents an 

the storage area 7 is to clearly and readily perform the ^ * ion of ^ two ^ images 52 and 53. 
subsequent retneval operations. Thus, information in commoT with the two capsule 

When the user needs the information with respect to ^ 52 and 53 can be retrieved in the subsequent 
the medieval ages he will place the selected capsule st ^ ^ when the ^ ^ holds the W0 M 
image m the position of year 1000 (see FIG 14 (a)). 40 sule ^ 52 md 53 m ^ ^ otfler with 

When the user 90 needs the mformaton m the range the right hand 8R ajjd the capsule image 50 representing 
from years 500 to 1500 he will place the left ^hand SL at United States" with the left hand SL, the OR operation 
the position of year 500 and the right hand 8R at the o(tht ^ ^ - ^ hM ^ ^ left ^ ^ 

E5?S ?^i 5 °° t0 desi ^t tC ^ C rangC ° f yearS < see «L and 8R can be represented. In other words the situa- 
FIG 14 (6)). When years on the time coordinate axis T 45 tion shown m FIG. 15 (5) represents a motion for re- 
are designated, the relevant historical events can be trieving a set of information in common with the cap- 
displayed in predetermined output formats. Thus, the sule 53 representing "Japan" and the capsule 
sequence of retneval operation is completed. image 52 representing "Germany" and information of 

When the user 90 needs to designate a range of years the capsule image 50 representing "United States", 

on the time coordinate axis T, he will retrieve informa- 50 of course, it is possible to define the motion for con- 

tion through three-dimensional image spaces being tacting the capsule images 52 and 53 against each other 

changed one after the other. To complete the retrieval as an OR operation and the motion for holding the 

operation, the user 90 can perform predetermined oper- capsule image 50 as an AND operation, 

ations as the sign, such as turning the hand(s), pushing a Thus, by two types of motions for holding a plurality 

particular capsule image, or holding a capsule image 55 of capsule images at a time, an AND operation and an 

representing "END." OR operation can be simply represented and a set of 

The above description focused on the retrieval opera- information to be retrieved can be readily designated 

tion of historical events. Further, for example, in the and changed. 

case for retrieving the evolutionary process of crea- Next, with reference to FIG. 16, a retrieval motion of 

tures, if the time coordinate axis T is present from the 60 an exclusive OR operation will be described, 

right to the left in the display space, when the user 90 In the case where the user 90 will select capsule im- 

extends the left hand to the left and the right hand to the ages representing "creatures", **plants", and "flowers" 

right, he can retrieve information in the past and in the in this sequence and place them in a storage area, he will 

future (predictable information). Moreover, in the case select a capsule image representing "trees" from 

where the time coordinate axis T is set in the direction 65 "plants" and place the selected capsule image in the 

of depth of the display space, when the operator 60 = storage area. Thereafter, the user 90 will pick a capsule 

performs hand motions such as moving the hand(s) image 61 representing "flowers" from the storage area 

inwardly or outwardly, he can straightforwardly and with the left hand 8L and pick a capsule image 63 repre- 
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senting "trees" with the right hand 8R. Thereafter, the Further, the motion patterns of the user's feet can be 
user 90 will contact the capsule image 63 representing detected. 

**trees" against the capsule image 61 representing "flow- Furthermore, the motion patterns of the user's hands 
ers" and then separate them. This motion represents an or the user's head can be detected by a TV camera 100 
exclusive OR operation. In other words, this motion 5 and the detected information can be sent to the arithme- 
represents **flowers" which do not blossom on "trees" tic operation unit 94. 

or "trees" which do not flower. In addition, a negation In FIG. 1, an example wherein a three-dimensional 
can be represented by a motion for holding a capsule image space is displayed before the eyes of the user 90 
image and then turning it. Thus, according to this em- by the three-dimensional display unit 91 having ski 
bodiment, by natural motions of human beings such as 10 goggle shape that the user 90 wears was described, 
contacting capsule images against each other, separat- However, as shown in FIG. 20, the user can wear a 
ing them, and turning them, the user 90 can retrieve private-eye 114 at a position slightly away from the 
information. head 110 and see a three-dimensional image space dis- 

FIG. 17 is a schematic diagram showing a retrieval played from the private-eye 114. When the private-eye 
method in accordance with another embodiment of the 15 114 is used, the sense of oppression against the head 110 
present invention. can be decreased. In addition, the shapes of capsule 

In the figure, a dictionary 70 with words alphabet!- images displayed in the three-dimensional image spaces 
cally arranged is displayed in a three-dimensional image can be freely designated in accordance with their mean- 
space. When the user 90 consults the dictionary 70, he ing and contents. 

can scroll screens by touching the surface of the dictio- 20 In FIGS. 1 and 2, an example where a three-dimen- 
nary 70, which is an image, with the hand 8R or 8L and sional image space was displayed by the three-dimen- 
leaf through the pages thereof. In this case, the user can sional display unit 91 that the user 90 wears was de- 
scroll the screens by knocking the left hand 8L and the scribed. However, instead of the three-dimensional 
right hand 8R. When there is a hit of word, phrase or display unit 91, the user can wear a two-dimensional 
topic, the user 90 can pick up an icon representing the 25 display unit 116 for displaying two-dimensional image 
word, phrase, or topic from the dictionary 70 so as to spaces. When the user 90 wears the two-dimensional 
directly retrieve the object display unit 116, the virtual left hand 8L, the virtual 

Since every word and every phrase of each page of right hand 8R, and the various capsule images 3, 4, 5, 6, 
the dictionary 70 has an embedded icon 72, the icon 72 and 7 are displayed in the two-dimensional image 
can function as an index of information like a capsule 30 spaces. 

image. Thus, when dictionaries in a variety of catego- With reference to FIG. 21, another embodiment of 
ries are provided and information is retrieved in this the information retrieval apparatus in accordance with 
way, since it is not necessary for the user 90 to carry the present invention will be described. In the figure, 
heavy dictionaries, he will not be fatigued from the capsule images (index display images) are displayed on 
information retrieval operation. 35 a flat display screen (display unit) 121 such as a CRT 

FIG. 18 shows a means for newly inputting a key- display unit. These capsule images 124, 125, and 126 
word which is not represented by a capsule image. An represent a set of indexes for controlling attributes of a 
input means 80 has a shape of a pen 81 which is dis- database. 

played in a space. The user 90 can hold the pen 81 with When the user contacts a touch pen 141 held in a real 
the virtual arm 8R or 8L. When the user 90 properly 40 right hand 140 against the capsule images 124, 125, and 
holds the pen 81, ink will discharge and thereby the user 126 displayed on the screen 121, information for per- 
90 can write characters in the space. When the user 90 forming predetermined operation patterns can be input 
attempts to create new keywords, relevant indexes for to the capsule images 124, 125, and 126. For example, 
controlling attributes of the database should have been when the user contacts the capsule image 125 with the 
already provided. 45 touch pen 141 and moves the touch pen 141 to the 

As described above, by utilizing the feature whereby position of the capsule image 124, the capsule images 
characters and/or geometries images can be written 125 and 124 can be superimposed, 
and/or drawn in the space, the information retrieval In other words, the flat display screen 121 is con- 
apparatus of the present invention can be used as a nected to the arithmetic operation unit 130. The arith- 
teaching tool for children. In this case, the present in- 50 metic operation unit 130 recognizes information which 
vention can provide most people from children to aged is input from the touch pen 141 and displays the motions 
people with interfaces to the computer graphics world. of the capsule images on the screen 121. In addition, the 
^ In addition, when a keyboard that any user lik es is arithmetic operation unit 130 performs set operations of 
*<Esplayed in the space and the user can input data~ffom indexes displayed with the capsule images 124, 125, and 
a unit which detects the above described hand (finger) 55 126 in accordance with the information being input 
motions, the user can select a favorite one from a vari- from the touch pen 141. The computed result of the 
ety of keyboards of different standards for use in input- arithmetic operation unit 130 is sent to the retrieval 
ting data. Thus, according to the present invention, the operation unit 131. The retrieval operation unit 131 
user can retrieve information by conventional keyboard inputs retrieval data into the database 132. 
operations that he is accustomed to. 60 In FIG. 21, when the user moves the capsule image 

In FIG. 1, an example wherein the motion patterns of 125 to the position of the capsule image 124 and super- 
the user's hands were detected by the data gloves 92R imposes them together, an AND operation of the cap- 
and 92L that he wears on his two hands and the motion sule images 124 and 125 Us represented. Thus, the arith- 
patterns were used as input information was described. metic operation unit 130 can perform the AND opera- 
However, it should be understood that besides the mo- 65 tion of the capsule images 124 and 125. Thus, the user 
tion patterns of the user's hands, the motion patterns of can retrieve information in common with each index 
the user's head 110 can be detected by a head band 112 displayed with the capsule images 124 and 125 from the 
which supports the three-dimensional display unit 91. database 132. 
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Next still another embodiment of the information 
retrieval apparatus in accordance with the present in- 
vention will be described with reference to FIG. 22. 

In FIG. 22, parts which correspond to parts shown in 
FIG. 21 are depicted by the same reference numerals, 5 
and detailed descripitions of them are omitted. 

As shown in FIG. 22, a righthand mouse 144 having 
a cord 144a is held by the real right hand 140, and the 
righthand mouse 144 is slid on an operation plate (not 
shown) so that information for performing predeter- 10 
mined operation patterns can be input to a capsule 
image 148. A lefthanded mouse 146 having a cord 146a 
is held by a real left hand 142, and the lefthand mouse 
146 is slid on the operation plate so that information for 15 
performing predetermined operation patterns can be 
input to a capsule image 150. In this way, for example, 
two capsule images 148 and 150 can be superimposed. 

In this case, the cord 144a of the righthand mouse 144 
and the cord 146a of the lefthand mouse 146 are con- 20 
nected to the arithmetic operation unit 130. The arith- 
metic operation unit 130 recognizes information which 
is input from the righthand mouse 144 and the lefthand 
mouse 146 and displays the motions of the capsule im- 
ages 148 and 150 on the screen 121. In addition, the 25 
arithmetic operation unit 130 performs set operations of 
indexes displayed with the capsule images 148 and 150 
in accordance with information input from the right- 
hand mouse 144 and the lefthand mouse 146. The com- 
puted result of the arithmetic operation unit 130 is sent 30 
to the retrieval operation unit 131. 

What is claimed is: 

1. An information retrieval apparatus, comprising: 

a three-dimensional display unit for displaying a set of 
indexes controlling attributes of a database in a 35 
three-dimensional image space in the sight of a user 
with index display images so as to allow the user to 
visually recognize said set of indexes; 

an input unit for detecting a motion pattern of the 
user's body against said index display images as 40 
input information and for displaying the motion 
patterns in said three-dimensional image space; and 

an arithmetic operation unit for recognizing the input 
information of the motion patterns received 
through said input unit, for selecting indexes dis- 
played with the corresponding index display im- 
ages in accordance with the motion patterns and 
for performing set logical operations of the se- 
lected indexes in accordance with the motion pat- 50 
terns so as to collate and retrieve information from 
said database, 

wherein the index display images can be moved in the 
three-dimensional image space in accordance with 
the motion patterns of the user's body, and the 55 
index information of the indexes vary in accor- 
dance with the positions of the index display im- 
ages in the three-dimensional image space. 

2. The information retrieval apparatus as set forth in 
claim 1, wherein said input unit is arranged to detect 60 
motion patterns of the user's hands as input information 
and to display the motion patterns of the hands in a 
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three-dimensional image space displayed by said three- 
dimensional display unit. 

3. The information retrieval apparatus as set forth in 
claim 1, wherein said input unit is arranged to detect 
motion patterns of the user's head as input information 
and to display the motion patterns of the heads in a 
three-dimensional image display space displayed by said 
three-dimensional display unit. 

4. The information retrieval apparatus as set forth in 
claim 1, wherein said three-dimensional display unit is 
arranged to display index display images on a time coor- 
dinate axis disposed in a three-dimensional image space, 
said arithmetic operation unit being arranged to per- 
form set operations of indexes displayed with predeter- 
mined index display images in a time sequence. 

5. The information retrieval apparatus as set forth in 
claim 1, wherein said input unit is a fitting member 
directly fitted on the user's body, said input unit being 
arranged to detect motion patterns of the body as input 
information. 

6. The information retrieval apparatus as set forth in 
claim 5, wherein said fitting member is a pair of data 
gloves fitted on the user's hands. 

7. The information retrieval apparatus as set forth in 
claim 6, wherein said data gloves have a pressure exert- 
ing mechanism for exerting a pressure to the user's 
hands in accordance with predetermined motion pat- 
terns thereof against index display images. 

8. The information retrieval apparatus as set forth in 
claim 1, wherein said input unit is a TV camera for 
detecting motion patterns of the user's body. 

9. A information retrieval apparatus, comprising: 

a three-dimensional display unit for displaying a set of 
indexes controlling attributes of a database in a 
three-dimensional image space in the sight of a user 
with index capsule images and a set of logical oper- 
ation means with operation capsule images so as to 
allow the user to visually recognize said set of 
indexes and said set of logical operation means; 

an input unit for detecting motion patterns of the 
user's body against said index capsule images and 
said operation capsule images as input information 
and for displaying the motion patterns in said three- 
dimensional image space; and 

an arithmetic operation unit for recognizing the input 
information of the motion patterns received 
through said input unit, for selecting indexes dis- 
played with the corresponding index capsule im- 
ages and logical operation means displayed with 
the corresponding operation capsule images in 
accordance with the motion patterns and for per- 
forming set logical operations of the selected in- 
dexes by using the selected logical operation means 
so as to collate and retrieve information from said 
database, 

wherein the index capsule images can be moved in 
the three-dimensional image space in accordance 
with the motion patterns of the user's body, and the 
index information of the indexes vary in accor- 
dance with the positions of the index display im- 
ages in the three-dimensional image space. 
***** 
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